54 research outputs found
What do Neural Machine Translation Models Learn about Morphology?
Neural machine translation (MT) models obtain state-of-the-art performance
while maintaining a simple, end-to-end architecture. However, little is known
about what these models learn about source and target languages during the
training process. In this work, we analyze the representations learned by
neural MT models at various levels of granularity and empirically evaluate the
quality of the representations for learning morphology through extrinsic
part-of-speech and morphological tagging tasks. We conduct a thorough
investigation along several parameters: word-based vs. character-based
representations, depth of the encoding layer, the identity of the target
language, and encoder vs. decoder representations. Our data-driven,
quantitative evaluation sheds light on important aspects in the neural MT
system and its ability to capture word structure.Comment: Updated decoder experiment
Can LLMs facilitate interpretation of pre-trained language models?
Work done to uncover the knowledge encoded within pre-trained language
models, rely on annotated corpora or human-in-the-loop methods. However, these
approaches are limited in terms of scalability and the scope of interpretation.
We propose using a large language model, ChatGPT, as an annotator to enable
fine-grained interpretation analysis of pre-trained language models. We
discover latent concepts within pre-trained language models by applying
hierarchical clustering over contextualized representations and then annotate
these concepts using GPT annotations. Our findings demonstrate that ChatGPT
produces accurate and semantically richer annotations compared to
human-annotated concepts. Additionally, we showcase how GPT-based annotations
empower interpretation analysis methodologies of which we demonstrate two:
probing framework and neuron interpretation. To facilitate further exploration
and experimentation in this field, we have made available a substantial
ConceptNet dataset comprising 39,000 annotated latent concepts
Scaled-up Discovery of Latent Concepts in Deep NLP Models
Pre-trained language models (pLMs) learn intricate patterns and contextual
dependencies via unsupervised learning on vast text data, driving breakthroughs
across NLP tasks. Despite these achievements, these models remain black boxes,
necessitating research into understanding their decision-making processes.
Recent studies explore representation analysis by clustering latent spaces
within pre-trained models. However, these approaches are limited in terms of
scalability and the scope of interpretation because of high computation costs
of clustering algorithms. This study focuses on comparing clustering algorithms
for the purpose of scaling encoded concept discovery of representations from
pLMs. Specifically, we compare three algorithms in their capacity to unveil the
encoded concepts through their alignment to human-defined ontologies:
Agglomerative Hierarchical Clustering, Leaders Algorithm, and K-Means
Clustering. Our results show that K-Means has the potential to scale to very
large datasets, allowing rich latent concept discovery, both on the word and
phrase level
What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysis
End-to-end DNN architectures have pushed the state-of-the-art in speech
technologies, as well as in other spheres of AI, leading researchers to train
more complex and deeper models. These improvements came at the cost of
transparency. DNNs are innately opaque and difficult to interpret. We no longer
understand what features are learned, where they are preserved, and how they
inter-operate. Such an analysis is important for better model understanding,
debugging and to ensure fairness in ethical decision making. In this work, we
analyze the representations trained within deep speech models, towards the task
of speaker recognition, dialect identification and reconstruction of masked
signals. We carry a layer- and neuron-level analysis on the utterance-level
representations captured within pretrained speech models for speaker, language
and channel properties. We study: is this information captured in the learned
representations? where is it preserved? how is it distributed? and can we
identify a minimal subset of network that posses this information. Using
diagnostic classifiers, we answered these questions. Our results reveal: (i)
channel and gender information is omnipresent and is redundantly distributed
(ii) complex properties such as dialectal information is encoded only in the
task-oriented pretrained network and is localised in the upper layers (iii) a
minimal subset of neurons can be extracted to encode the predefined property
(iv) salient neurons are sometimes shared between properties and can highlights
presence of biases in the network. Our cross-architectural comparison indicates
that (v) the pretrained models captures speaker-invariant information and (vi)
the pretrained CNNs models are competitive to the Transformers for encoding
information for the studied properties. To the best of our knowledge, this is
the first study to investigate neuron analysis on the speech models.Comment: Submitted to CSL. Keywords: Speech, Neuron Analysis,
Interpretibility, Diagnostic Classifier, AI explainability, End-to-End
Architectur
- …